Geometrical Singularities in the Neuromanifold of Multilayer Perceptrons
نویسندگان
چکیده
Singularities are ubiquitous in the parameter space of hierarchical models such as multilayer perceptrons. At singularities, the Fisher information matrix degenerates, and the Cramer-Rao paradigm does no more hold, implying that the classical model selection theory such as AIC and MDL cannot be applied. It is important to study the relation between the generalization error and the training error at singularities. The present paper demonstrates a method of analyzing these errors both for the maximum likelihood estimator and the Bayesian predictive distribution in terms of Gaussian random fields, by using simple models.
منابع مشابه
Singularities Affect Dynamics of Learning in Neuromanifolds
The parameter spaces of hierarchical systems such as multilayer perceptrons include singularities due to the symmetry and degeneration of hidden units. A parameter space forms a geometrical manifold, called the neuromanifold in the case of neural networks. Such a model is identified with a statistical model, and a Riemannian metric is given by the Fisher information matrix. However, the matrix ...
متن کاملGeometrical Initialization, Parametrization and Control of Multilayer Perceptrons : Application to Function Approximation 1
| This paper proposes a new method to reduce training time for neural nets used as function approximators. This method relies on a geometrical control of Multilayer Perceptrons (MLP). A geometrical initializa-tion gives rst better starting points for the learning process. A geometrical parametriza-tion achieves then a more stable convergence. During the learning process, a dynamic geometrical c...
متن کاملGeometrical Initialization, Parametrization and Control of Multilayer Perceptrons: Application to Function Approximation
This paper proposes a new method to reduce training time for neural nets used as function approximators. This method relies on a geometrical control of Multilayer Perceptrons (MLP). A geometrical initialization gives first better starting points for the learning process. A geometrical parametrization achieves then a more stable convergence. During the learning process, a dynamic geometrical con...
متن کاملDynamics of Learning Near Singularities in Layered Networks
We explicitly analyze the trajectories of learning near singularities in hierarchical networks, such as multilayer perceptrons and radial basis function networks, which include permutation symmetry of hidden nodes, and show their general properties. Such symmetry induces singularities in their parameter space, where the Fisher information matrix degenerates and odd learning behaviors, especiall...
متن کاملSingularities in Learning Models: Gaussian Random Field Approach
Singularities are ubiquitous in the parameter space of hierarchical models such as multilayer perceptrons. At singularities, the Fisher information metric degenerates, implying that the Cramér-Rao paradigm does no more hold and the classical model selection theory such as AIC and MDL cannot be applied. It is important to study the relation between the generalization error and the training error...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2001